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Principal  Investigator:  Professor  Edward  K,  Blum,  Mathematics 
Department,  University  of  Southern  California,  Los  Angeles,  CA 
90089-1113. 

Title  of  Project:  Mathematical  and  Numerical  Analysis  Aspects  of 
Quasi-Neural  Networks. 

Significant  results  were  obtained  on  feed-forward  and  recurrent 
networks.  Five  technical  papers  were  accepted  for  publication  in 
leading  journals. 

1.  Feed-forward  networks.  The  P.I.  and  his  Ph.D,  student, 
Leong  Li,  obtained  a  basic  result  on  feed-forward  networks.  This  is 
described  in  the  paper  "Approximation  theory  and  feed-forward 
networks",  which  has  been  published  in  the  journal.  Neural  Networks 
(1991)  4,  511-515.  The  basic  result  is  that  feed-forward  networks 
of  McCuIloch-Pitts  neurons  yield  arbitrarily  close  piecewise-constant 
approximations  to  continuous  and  L2-functions.  The  fundamental 
idea  of  piecewise-constant  approximation  was  not  recognized  by 
previous  researchers.  By  using  this  idea,  Blum  and  Li  were  able  to 
give  a  simple  technique  for  constructing  3-layer  networks  to 
uniformly  approximate  arbitrary  real  continuous  functions  of  n 
variables  and  also  arbitrary  L2-functions  in  the  mean  square  norm 
with  arbitrary  accuracy.  This  technique  should  be  useful  in  many  of 
the  applications  of  feed-forward  networks.  Li  has  also  published  a 
second  paper  in  this  topic. 

2.  Dynamical  behavior  of  recurrent  neural  networks. 
Recurrent  neural  networks,  that  is,  networks  with  feedback  loops, 
exhibit  all  the  phenomena  found  in  classical  dynamical  systems 
governed  by  systems  of  ordinary  differential  equations  (ODE's).  In 
fact,  if  the  neuron  activities  are  modeled  by  ODE's,  then  the  network 
is  modeled  as  a  system  of  coupled  ODE's.  However,  it  is  also  of 


interest  to  model  the  neurons  as  analog  elements  which  have  states 
(outputs)  in  the  real  interval  /  =[0,1]  governed  by  an  input-output 
function  of  sigmoidal  type  and  operating  in  discrete-time.  If  the 
operation  is  synchronous,  then  the  network  is  modeled  as  a  system 
of  difference  equations  rather  than  differential  equations.  Thus,  the 
dynamics  is  that  of  interation  of  a  map  F  according  to  the 

general  equation  x  (t  +1)=F  {x  (t )),  t  =0,1,2,...,  where  jc  (O  is  the  state 
vector  at  lime  t .  In  many  application  (e.g.,  associative  memories, 
oscillators  of  various  kinds,  etc.),  F  has  the  form  F  (x  )  =  a  (W  x  +  L), 
where  IV  is  a  real  "weight”  matrix,  L  is  the  "threshold"  vector  and 
sigma  is  a  sigmoidal-type  nonlinear  vector  function.  In  these 
applications,  one  is  interested  in  the  existence  and  stability  of  fixed- 
points  and  periodic  orbits  in  phase  space  (i.e.  oscillations),  There  is  a 
natural  parameter  in  such  networks,  namely,  the  "gains"  of  the 
neurons,  as  represented  by  the  maximum  slopes,  a  i,  of  the 
sigmoidal  functions.  One  is  also  interested  in  the  changes  in  the 
dynamics  induced  by  variations  in  the  gains,  in  particular,  in  the 
occurrence  of  bifurcation  points  at  which  stable  equilibria  become 
unstable  or  at  which  stable  oscillations  appear  and  disappear. 

The  P.I,  and  his  Ph.D,  student,  Xin  Wang,  have  studied  the  dynamics 
of  such  networks  and  obtained  a  number  of  new  results  which 
should  be  useful  in  applications.  These  are  described  in  detail  in  the 
paper  "Stability  of  Fixed  Points  and  Periodic  Orbits  and  Bifurcations 
in  Analog  Neural  Networks",  accepted  for  publication  in  the  journal. 
Neural  Networks. 

The  case  of  symmetric  W  has  received  attention  in  the  recent 
literature.  We  consider  the  simplest  instance  of  such  a  network 
having  only  2  neurons  coupled  to  operate  like  an  analog 
multivibrator  circuit.  We  show  that  the  dynamics  of  fixed  points  and 
periodic  orbits  (oscillations)  can  be  completely  characterized  by 
elementary  analysis  without  introducing  a  Liapunov  function,  the 
usual  tool.  This  analysis  can  be  extended  to  special  networks  built 
from  these  2-neuron  modules.  The  same  methods  of  analysis  are 
then  applied  to  non-symmetric  W  .  For  example,  it  is  shown  how  to 


construct  ring  oscillators  of  any  period.  The  case  of  coupled 
oscillators  and  forced  oscillators  is  also  subjected  to  the  same 
analysis.  By  varying  the  gains  of  the  sigmoidal  neurons  we  obtain 
interesting  bifurcations  into  oscillatory  behavior. 

3.  Chaotic  dynamics  in  neural  networks.  It  is  known  that 
neural  networks  are  capable  of  chaotic  behavior.  New  results  are 
reported  in  the  paper,  "Period-doublings  to  Chaos  in  a  Simple  Neural 
Network:  A  Mathematical  Proof  by  Xin  Wang  (Ph.D.  student  of  the 
P.I.).  Usually,  chaotic  dynamics  is  demonstrated  by  simulation.  In 
this  part  of  the  project,  Wang  was  able  to  prove  that  chaos  occurs  in 
a  2-neuron  network  with  self-loops.  Thus,  this  simple  network 
differs  from  the  2-neuron  network  studied  by  Blum  and  Wang 
described  above.  However,  the  dynamics  is  again  discrete-time 
iteration  of  a  nonlinear  sigmoidal  map.  By  restricting  the  weight 
matrix, W  to  be  singular,  it  is  possible  to  reduce  the  dynamics  to 
iteration  of  a  family  of  1 -dimensional  maps  parametrized  by  the 
gain,  p  of  the  neurons.  Appealing  to  the  known  theory  of  1- 
dimensional  maps,  Wang  is  able  to  prove  that  the  2-neuron  networks 
with  singular  W  follow  the  period-doubling  route  to  chaos  as  p 
increases.  To  illustrate  the  theoretical  result,  Wang  computes  the 
bifurcation  diagrams  for  two  networks  with  different  W  's  and 
demonstrates  the  occurrence  of  chaos  as  p  increases. 

4.  Discrete-time  versus  continuous-time  dynamics.  Blum 
and  Wang  compare  the  dynamical  properties  of  continuous-time 
neural  networks  defined  by  systems  of  differential  equations  of  the 
leaky  integrator  type  with  the  dynamical  properties  of  the 
approximating  difference  equations.  Quantitative  results  in  the  step- 
size  of  the  discrete-time  numerical  method  to  guarantee  the  same 
dynamics  are  obtained. 
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